<!DOCTYPE html>
<html>
<head>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><meta charset="utf-8">
  
  <title>CRF Layer on the Top of BiLSTM - 4 | CreateMoMo</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  <meta name="description" content="2.4 Real path scoreIn section 2.3, we supposed that every possible path has a score $ P_{i} $ and there are totally $ N $ possible paths, the total score of all the paths is $ P_{total} = P_1 + P_2 +">
<meta property="og:type" content="article">
<meta property="og:title" content="CRF Layer on the Top of BiLSTM - 4">
<meta property="og:url" content="http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/index.html">
<meta property="og:site_name" content="CreateMoMo">
<meta property="og:description" content="2.4 Real path scoreIn section 2.3, we supposed that every possible path has a score $ P_{i} $ and there are totally $ N $ possible paths, the total score of all the paths is $ P_{total} = P_1 + P_2 +">
<meta property="og:locale" content="default">
<meta property="og:updated_time" content="2017-11-11T00:20:21.127Z">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="CRF Layer on the Top of BiLSTM - 4">
<meta name="twitter:description" content="2.4 Real path scoreIn section 2.3, we supposed that every possible path has a score $ P_{i} $ and there are totally $ N $ possible paths, the total score of all the paths is $ P_{total} = P_1 + P_2 +">
  
  
    <link rel="icon" href="/favicon.png">
  
  
    <link href="//fonts.googleapis.com/css?family=Source+Code+Pro" rel="stylesheet" type="text/css">
  
  <link rel="stylesheet" href="/css/style.css"><!-- hexo-inject:begin --><!-- hexo-inject:end -->
  

</head>

<body>
  <!-- hexo-inject:begin --><!-- hexo-inject:end --><div id="container">
    <div id="wrap">
      <header id="header">
  <div id="banner"></div>
  <div id="header-outer" class="outer">
    <div id="header-title" class="inner">
      <h1 id="logo-wrap">
        <a href="/" id="logo">CreateMoMo</a>
      </h1>
      
    </div>
    <div id="header-inner" class="inner">
      <nav id="main-nav">
        <a id="main-nav-toggle" class="nav-icon"></a>
        
          <a class="main-nav-link" href="/">Home</a>
        
          <a class="main-nav-link" href="/archives">Archives</a>
        
      </nav>
      <nav id="sub-nav">
        
        <a id="nav-search-btn" class="nav-icon" title="Search"></a>
      </nav>
      <div id="search-form-wrap">
        <form action="//google.com/search" method="get" accept-charset="UTF-8" class="search-form"><input type="search" name="q" class="search-form-input" placeholder="Search"><button type="submit" class="search-form-submit">&#xF002;</button><input type="hidden" name="sitesearch" value="http://createmomo.github.io"></form>
      </div>
    </div>
  </div>
</header>
      <div class="outer">
        <section id="main"><article id="post-CRF-Layer-on-the-Top-of-BiLSTM-4" class="article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/" class="article-date">
  <time datetime="2017-10-17T23:03:39.000Z" itemprop="datePublished">2017-10-17</time>
</a>
    
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 class="article-title" itemprop="name">
      CRF Layer on the Top of BiLSTM - 4
    </h1>
  

      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h4 id="2-4-Real-path-score"><a href="#2-4-Real-path-score" class="headerlink" title="2.4 Real path score"></a>2.4 Real path score</h4><p>In section 2.3, we supposed that every possible path has a score $ P_{i} $ and there are totally $ N $ possible paths, the total score of all the paths is $ P_{total} = P_1 + P_2 + … + P_N = e^{S_1} + e^{S_2} + … + e^{S_N} $, $ e $ is the mathematical constant $ e $. </p>
<p>Obviously, there must be a path is the real one among all the possible paths. For exmaple, the real path of the sentence in section 1.2 is <strong>“START B-Person I-Person O B-Organization O END”</strong>. The others are incorrect such as “START B-Person B-Organization O I-Person I-Person B-Person”. $ e^{S_i} $ is the score of $ i^{th} $ path.</p>
<p>During the training process, the crf loss function only need two scores: the score of the real path and the total score of all the possbile paths. <strong>The proportion of the real path score among the scores of all the possible paths will be increased gradually.</strong></p>
<p>The calculation of a real path score, $e^{S_i}$, is very straightforward. </p>
<a id="more"></a>
<p>Here we focus on the calculation of $ S_i $.</p>
<p>Take the real path, <strong>“START B-Person I-Person O B-Organization O END”</strong>, we used before, for example:</p>
<ul>
<li>We have a sentence which has 5 words, $w_1, w_2, w_3, w_4, w_5$</li>
<li>We add two more extra words which denote the start and the end of a sentence, $w_0, w_6$</li>
<li>$S_i$ consists of 2 parts: $S_i = EmissionScore + TransitionScore $ (The emission and transition score are expanined in section 2.1 and 2.2)</li>
</ul>
<p><strong>Emission Score:</strong><br>$EmissionScore=x_{0,START}+x_{1,B-Person}+x_{2,I-Person}+x_{3,O}+x_{4,B-Organization}+x_{5,O}+x_{6,END}$</p>
<ul>
<li><p>$ x_{index,label} $ is the score if the $index^{th}$ word is labelled by $ label $</p>
</li>
<li><p>These scores $ x_{1,B-Person} $ $ x_{2,I-Person} $ $ x_{3,O} $  $ x_{4,Organization} $  $ x_{5,O} $ are from the previous BiLSTM output.</p>
</li>
<li><p>As for the $ x_{0,START} $ and $ x_{6,END} $, we can just set them zeros.</p>
</li>
</ul>
<p><strong>Transition Score:</strong><br>$TransitionScore=$<br>$t_{START-&gt;B-Person} + t_{B-Person-&gt;I-Person} + $<br>$t_{I-Person-&gt;O} + t_{0-&gt;B-Organization} + t_{B-Organization-&gt;O} + t_{O-&gt;END}$</p>
<ul>
<li>$t_{label1-&gt;label2}$ is the transition score from $label1$ to $label2$</li>
<li>These scores come from the CRF Layer. In other words, these transition scores are actually the parameters of CRF Layer.</li>
</ul>
<p>To sum up, now we can calculate $S_i$ and as well as the path score $e^{S_i}$. The next step is <strong>how to calculate the total score of all the possible paths?</strong></p>
<h3 id="Next"><a href="#Next" class="headerlink" title="Next"></a>Next</h3><h4 id="2-5-The-total-score-of-all-the-possible-paths"><a href="#2-5-The-total-score-of-all-the-possible-paths" class="headerlink" title="2.5 The total score of all the possible paths"></a>2.5 The total score of all the possible paths</h4><p>How to calculate the total score of all the possible paths of a sentence with a step-by-step toy example.</p>
<p>This section would be one of the most important and a bit difficult part. But DO NOT worry. The toy example given in this section will explain the details as simple as possible.</p>
<p><strong>(Sorry for my late update, I will try my best to squeeze time for updating the following sections.)</strong></p>
<h2 id="References"><a href="#References" class="headerlink" title="References"></a>References</h2><p>[1]  Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K. and Dyer, C., 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.<br><a href="https://arxiv.org/abs/1603.01360" target="_blank" rel="external">https://arxiv.org/abs/1603.01360</a></p>
<blockquote>
<p>When you reprint or distribute this article, please include the original link address.</p>
</blockquote>

      
    </div>
    <footer class="article-footer">
      <a data-url="http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/" data-id="ck0lc6yz40002ucp03u60hg7a" class="article-share-link">Share</a>
      
        <a href="http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/#disqus_thread" class="article-comment-link">Comments</a>
      
      
    </footer>
  </div>
  
    
<nav id="article-nav">
  
    <a href="/2017/11/11/CRF-Layer-on-the-Top-of-BiLSTM-5/" id="article-nav-newer" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Newer</strong>
      <div class="article-nav-title">
        
          CRF Layer on the Top of BiLSTM - 5
        
      </div>
    </a>
  
  
    <a href="/2017/10/08/CRF-Layer-on-the-Top-of-BiLSTM-3/" id="article-nav-older" class="article-nav-link-wrap">
      <strong class="article-nav-caption">Older</strong>
      <div class="article-nav-title">CRF Layer on the Top of BiLSTM - 3</div>
    </a>
  
</nav>

  
</article>


<section id="comments">
  <div id="disqus_thread">
    <noscript>Please enable JavaScript to view the <a href="//disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>
  </div>
</section>
</section>
        
          <aside id="sidebar">
  
    

  
    

  
    
  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Archives</h3>
    <div class="widget">
      <ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/07/">July 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2019/01/">January 2019</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2018/01/">January 2018</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/12/">December 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/11/">November 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/10/">October 2017</a></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2017/09/">September 2017</a></li></ul>
    </div>
  </div>


  
    
  <div class="widget-wrap">
    <h3 class="widget-title">Recent Posts</h3>
    <div class="widget">
      <ul>
        
          <li>
            <a href="/2019/07/18/Table-of-Contents/">Table of Contents</a>
          </li>
        
          <li>
            <a href="/2019/01/07/Probabilistic-Graphical-Models-Revision-Notes/">Probabilistic Graphical Models Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/23/Super-Machine-Learning-Revision-Notes/">Super Machine Learning Revision Notes</a>
          </li>
        
          <li>
            <a href="/2018/01/17/My-Life/">My Life</a>
          </li>
        
          <li>
            <a href="/2017/12/07/CRF-Layer-on-the-Top-of-BiLSTM-8/">CRF Layer on the Top of BiLSTM - 8</a>
          </li>
        
      </ul>
    </div>
  </div>

  
</aside>
        
      </div>
      <footer id="footer">
  
  <div class="outer">
    <div id="footer-info" class="inner">
      &copy; 2019 CreateMoMo<br>
      Powered by <a href="http://hexo.io/" target="_blank">Hexo</a>
    </div>
  </div>
</footer>
    </div>
    <nav id="mobile-nav">
  
    <a href="/" class="mobile-nav-link">Home</a>
  
    <a href="/archives" class="mobile-nav-link">Archives</a>
  
</nav>
    
<script>
  var disqus_shortname = 'createmomo';
  
  var disqus_url = 'http://createmomo.github.io/2017/10/17/CRF-Layer-on-the-Top-of-BiLSTM-4/';
  
  (function(){
    var dsq = document.createElement('script');
    dsq.type = 'text/javascript';
    dsq.async = true;
    dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
    (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
  })();
</script>


<script src="//ajax.googleapis.com/ajax/libs/jquery/2.0.3/jquery.min.js"></script>


  <link rel="stylesheet" href="/fancybox/jquery.fancybox.css">
  <script src="/fancybox/jquery.fancybox.pack.js"></script>


<script src="/js/script.js"></script>

  </div>
<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        tex2jax: {
            inlineMath: [ ["$","$"], ["\\(","\\)"] ],
            skipTags: ['script', 'noscript', 'style', 'textarea', 'pre', 'code'],
            processEscapes: true
        }
    });
    MathJax.Hub.Queue(function() {
        var all = MathJax.Hub.getAllJax();
        for (var i = 0; i < all.length; ++i)
            all[i].SourceElement().parentNode.className += ' has-jax';
    });
</script>
<!-- <script src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>-->
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-MML-AM_CHTML"></script><!-- hexo-inject:begin --><!-- hexo-inject:end -->
</body>
</html>